1. Technical Field
The present invention relates to an image pickup device, an image pickup system, an image pickup method, an image processing device, and an image pickup method, which are capable of reducing the amount of HDR image data in a state of ensuring a relatively wide gradation range and dynamic range.
2. Related Art
With many pixels and miniaturization of an image pickup device, a pixel size (sensor cell) is miniaturized and, as a result, the amount of saturation charges is reduced and a dynamic range (D range) narrows.
In order to compensate for narrowness of the D range of the image pickup device, an image pickup device capable of performing photographing at two types or more of exposure times in one frame period is suggested. In an image pickup device in which two fields configure one frame, an image pickup method for changing an exposure time by an odd-numbered or even-numbered field is also suggested and thus wide D range photographing is possible. In particular, in a complementary metal oxide semiconductor (CMOS) sensor, plural numbers of times of sampling or multiple channels are possible in an image pickup period of one frame by the research on a circuit configuration. According to a nondestructive readable sensor, plural numbers of times of sampling are facilitated.
Images photographed at two types or more of exposure times (for example, a standard exposure time, a short exposure time or the like) are synthesized by an external unit or an internal unit of the sensor, thereby obtaining a wide D range image. In a known image display device, from a limitation of a contrast ratio which can be expressed, tone mapping for reducing the information amount by 50 dB or 8 bit is performed. Since the information amount of the image of the wide D range is reduced into the limited dynamic range (D range), an image having a low contrast ratio (which is different from an image which can be viewed by a person) is obtained.
As a technology of compressing an image according to the capability of an output device, for example, there is an image pickup device described in JP-A-2000-188717.
In a computer field such as a personal computer (PC), a computer graphic (CG) technology for modeling a three-dimensional shape, the reflectivity and the color tone of a subject or a light source, virtually deciding the position of the light source and the position of a viewing point (camera), and determining how a two-dimensional image is viewed from the viewing point by computation is used in movie, design and game fields. The CG technology can treat an infinite D range by a high dynamic range (HDR) image for representing an image to a floating-point and can represent a natural image close to an image viewed by a personal without overexposure or underexposure. In general, in order to generate the HDR image, expensive computation resources, dedicated software, programming technologies, vast computation resources and time are necessary.
In a difference between the former (image pickup device) and the latter (CG technology) in generation of an image (formation of a picture), the former (image pickup device) forms and records a picture in consideration of the D range of the output device in the device (since the D range is fixed, a fixed-point process is used). In the former, gradation information is lost in order to compress contrast information of brightness and, as a result, if the recorded image is secondarily used, for example, if the image is displayed in a display device or is printed, the image cannot be sufficiently reproduced at the output device. In contrast, in the CG technology, since the image is generated by a floating-point process, the contrast information is not lost. Accordingly, if the image is secondarily used, a final image can be generated on the basis of the capability or the characteristics of the display device or the visibility of a user and more natural sensitive image (close to the sensitivity of a person) can be expressed.
In a floating-point arithmetic operation of physical simulation, since an arithmetic operation result of any time is used in an arithmetic operation at a next time, accumulation of operation errors may be problematic when a system performs an objective operation. In a three-dimensional CG technology, if a viewing point is changed on the basis of any rule, arithmetic operation errors are accumulated and thus an image may be damaged.
Meanwhile, if image processing is performed while photographing a nature image, since information (image) to be processed is always updated, a problem such as the accumulation of the arithmetic operation errors is not caused. In contrast, a problem such as overexposure or underexposure due to a narrow D range causes lack of pixel information and thus may be problematic when the system performs an objective operation. In particular, in the overexposure, it cannot be determined whether it is an image or a defect in machine vision and an operation different from the object is performed by the system. In the machine vision, since the image is globally (macro) grasped in area detection (labeling), edge detection, correlation detection, binarization process, the system capability such as recognition depends on the information loss such as overexposure or underexposure, rather than arithmetic operation errors of the pixel value. That is, in the machine vision, rather than reducing of arithmetic operation errors by the fixed-point arithmetic operation, it is important that a fixed point is converted into a floating-point so as to adjust the D range to be close to the characteristics of person's eyes (high-precision), thereby obtaining a sufficient contrast ratio.
However, in the technology of JP-A-2000-188717, since the arithmetic operation process of the image pickup device is performed by the fixed-point arithmetic operation, if the bit number of an image treated is increased, the number of arithmetic devices used in the arithmetic operation is exponentially increased as the bit number is increased. Thus, a circuit scale or circuit cost may be remarkably increased.
Since the output of the image pickup device becomes image data of which the pixel values are expressed by an integer, contrast information may be lost.